\section{Evaluation}
\label{sec:eval}

Our previously stated goal was to build intelligent agents for playing othello that could defeat weaker agents, such as ourselves, with efficiency and regularity.  To this end we have done designed and run a tournament between 19 of our 38 agents.

The precedent for running Othello computer tournaments [paderborn tournmaent cite] is to first have a round-robin approach where each player plays every other player one time.  From this round a seeding of the players is determined based on their number of wins and their average margin of winning.  Then a bracketed tournament is run between the players until a winner is determined.

As we have 19 agents for this tournament, it was infeasible for us to run 19! games for a round-robin round in the our time constraints.  We have substituted this round-robin for a series of ??? games against a random player under the assumption that a better player will more regularly defeat a random player.  The results can be seen in Table~\ref{tbl:rrresults}.  

From this set of games we established a seeding of the top 12 agents based on the number of games they won and the total margin of victory for tie breakers.  For each bracket agents played a set of 6 games against one another with ties broken by the total margin of victory.  In the final round the three finalist agents played a series of 6 games against one another, with the Mobility-Stability heuristic with $\alpha\beta$ pruning coming out on top.  The full results can be seen in Table~\ref{tbl:fresults}.

\begin{table}
\begin{centering}
\begin{tabular}{|l|l|l|c|c|c|} 
Name (seed) & Search & Heuristic & Wins & Losses & Ties \\ \hline \hline
naive				& ???    & Naive 				& 5 & 1 & 0 \\ \hline
greedyscore			& ???	 & Greedy Score 			& 4 & 2 & 0 \\ \hline
greedysmartscore		& ???    & Greedy Smart Score	& 6 & 0 & 0 \\ \hline
greedymob			& ???    & Greedy Mobility 		& 4 & 1 & 1 \\ \hline
greedystab (10)		& ???    & Greedy Stability 		& 5 & 0 & 1 \\ \hline
scoremax (12)		& ???    & Scoremax 				& 6 & 0 & 0 \\ \hline
smartscoremax (4)	& ???    & smartscoremax	    		& 6 & 0 & 0 \\ \hline
mobimax			 	& ???    & mobimax				& 2 & 2 & 2 \\ \hline
mobiplus 		 	& ???	 & mobiplus				& 5 & 1 & 0 \\ \hline
stabmax 	(8)			& ???	 & stabmax				& 6 & 0 & 0 \\ \hline
mobstab	(7)			& ???    & mobstab				& 6 & 0 & 0 \\ \hline
most (2)				& ???	 & most					& 6 & 0 & 0 \\ \hline
scoreab 	(3)			& ???	 & scoreab				& 6 & 0 & 0 \\ \hline
smartscoreab (11)	& ???    & smartscoreab			& 6 & 0 & 0 \\ \hline
mobab				& ???    & mobab					& 3 & 2 & 1 \\ \hline
mobplusab (6)		& ???	 & mobplusab				& 6 & 0 & 0 \\ \hline
stabab (5)			& ???    & stabab				& 5 & 1 & 0 \\ \hline
mobstabab (1)		& ???    & mobstabab				& 6 & 0 & 0 \\ \hline
mostab (7)			& ???    & mostab				& 6 & 0 & 0 \\ \hline
\end{tabular}
\caption{Results of a heuristic and search combination agent against a random agent in a 6 game match.}
\label{tbl:rrresults}
\end{centering}
\end{table}



\begin{table}
\begin{centering}

\begin{tabular}{|l|l|l|l|l|}
Seed & Name & Round 1 & Round 2 & Winner \\ \hline
1 	 & mobstabab & \multirow{2}{*}{1 mobstabab (6-0)} & \multirow{4}{*}{1 mobstabab (6-0)} & \multirow{12}{*}{1 mobstabab (5-1)(5-1)} \\ \cline{1-2}
12 	 & scoremax  &  &	& 	 \\ \cline{1-3}
2 	 & most		 & \multirow{2}{*}{11 smartscoreab (6-0)} & &  \\ \cline{1-2}
11   & smartscoreab &   & &   \\ \cline{1-4}
3 	 & scoreab & \multirow{2}{*}{3 scoreab (3-3), tie: 106-40} & \multirow{4}{*}{4 smartscoremax (5-1)} &  \\ \cline{1-2}
10 	 & greedystab  &  &	& 	 \\ \cline{1-3}
4 	 & smartscoremax& \multirow{2}{*}{4 smartscoremax (5-1)} & &  \\ \cline{1-2}
9    & mostab &   &    & \\ \cline{1-4}
5 	 & stabab & \multirow{2}{*}{5 stabab (5-1)} & \multirow{4}{*}{5 stabab (3-3), tie: 128-60} &  \\ \cline{1-2}
8 	 & stabmax  &  &	& 	 \\ \cline{1-3}
6 	 & mobplusab & \multirow{2}{*}{7 mobstab (5-1)} & &  \\ \cline{1-2}
7    & mobstab &   &    & \\ \hline
%1 & mobstabab & \multirow{2}{*}{mobstabab (6-0)} & & & \\
%12 & scoremax &  	  & 		&		 &  \\
\end{tabular}
\caption{Round by round results of the seeded players having 6 game matches.}
\label{tbl:fresults}
\end{centering}
\end{table}

%What was our experimental design, what did we expect to see?
%Why did these methodologies make sense? 

\input{eval_search}
\input{eval_heu}